weight activation training-reduce memory
Sparse Weight Activation Training- Reduce memory and training time in Machine Learning
A little bit ago, I covered Google AI's pathways architecture, calling it a revolution in Machine Learning. One of the standouts in Google's novel approach was the implementation of sparse activation in their training architecture. I liked this idea so much that I decided to explore this in a lot more depth. That's where I came across Sparse Weight Activation Training (SWAT), by some researchers at the Department of Electrical And Computer Engineering, University of British Columbia. And the paper definitely has me excited.